Papers with data mixing
Dancing in Chains: Reconciling Instruction Following and Faithfulness in Language Models (2024.emnlp-main)
Copied to clipboard
Zhengxuan Wu, Yuhao Zhang, Peng Qi, Yumo Xu, Rujun Han, Yian Zhang, Jifan Chen, Bonan Min, Zhiheng Huang
| Challenge: | Modern language models fail to follow human instructions while being faithful . a trade-off exists between instruction following and faithfulness when training LMs . |
| Approach: | They propose a method that relies on Reject-sampling by Self-instruct with Continued Fine-tuning to train LMs to follow human instructions while being faithful. |
| Outcome: | The proposed method outperforms vanilla MTL with high-quality data, but with significantly smaller data. |
TiKMiX: Efficient Semi-Dynamic Data Mixture via Data Influence for LLM Pre-training (2026.acl-long)
Copied to clipboard
Yifan Wang, null Binbinliu, Fengze Liu, Yuanfan Guo, Jiyao Deng, Xuecheng Wu, Weidong Zhou, Xiaohuan Zhou, Taifeng Wang
| Challenge: | Static data mixing strategies in large language models are often suboptimal as they fail to adapt to the model’s evolving learning states. |
| Approach: | They propose a semi-dynamic data mixing framework that uses a key observation of influence ranking invariance to reduce computational overhead by 80% . |
| Outcome: | The proposed method reduces computational overhead by 80% and achieves an average performance gain of 2% across nine downstream benchmarks, effectively mitigating data under-digestion. |
CAARMA: Class Augmentation with Adversarial Mixup Regularization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Speaker verification tasks require inference of unseen classes using specialized losses. |
| Approach: | They propose a class augmentation framework that generates synthetic classes through data mixing in the embedding space. |
| Outcome: | The proposed framework improves speaker verification tasks by 8% over baseline models. |
CommonIT: Commonality-Aware Instruction Tuning for Large Language Models via Data Partitions (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current studies have focused on fine-tuning, but the use of instruction tuning is not as effective as fine-cuning. |
| Approach: | They propose a commonality-aware instruction tuning strategy to cluster instruction datasets into distinct groups with three proposed metrics Task, Embedding and Length. |
| Outcome: | The proposed strategy boosts an average improvement of 2.1% on the general domain and 5.2% on the special domain. |
UniGeM: Unifying Data Selection and Mixing via Geometric Exploration and Mining (2026.findings-acl)
Copied to clipboard
Changhao Wang, null Yunfeiyu, Xinhao Yao, Jiaolong Yang, Lu Yu, Junpeng Fang, Chaobo Li, Riccardo Cantoro, Qing Cui, Jun Zhou
| Challenge: | Large Language Models (LLMs) scaling is limited by data quality and domain mixing and instance selection are two separate problems. |
| Approach: | They propose a framework that unifies mixing and selection without training proxy models or relying on external reference datasets. |
| Outcome: | The proposed framework achieves 2.0 data efficiency over a random baseline and further improves overall performance compared to SOTA methods in reasoning-heavy evaluations and multilingual generalization. |
DEM: Distribution Edited Model for Training with Mixed Data Distributions (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent fine-tuning approaches for large language models require supervised finetun on diverse datasets and follow different distributions. |
| Approach: | They propose a distribution edited model that integrates models individually trained on each data source with the base model using basic element-wise vector operations. |
| Outcome: | The proposed model outperforms baseline models on a variety of benchmarks and is cheaper than standard data mixing methods. |